• Two aspects of this type of utility, intrinsic satisfaction from behaving in accordance with a norm and extrinsic incentives to comply.

• Gerber, Green, and Larimer (2008) test intrinsic motives in a large scale field experiment by applying varying degrees of extrinsic pressure on voters using series of mailings to 180,002 households before the August 2006 primary election in Michigan.

  • \(Y_i\): Voted in the primary election (Outcome)
  • \(T_i\): Type of mailing (Treatment)

• Randomization (complete or simple) implies

\[ \{Y_i(1), Y_i(0)\} \ {\mbox{$\perp\!\!\!\perp$}}\ T_i \]

• Identification Question: Can we write down \(\tau_{ATE}\) only with observed data (\(Y_i, T_i\))?

• Estimation Question: Can we estimate \({\mathbb{E}}[Y_i {\:\vert\:}T_i = 1] - {\mathbb{E}}[Y_i {\:\vert\:}T_i = 0]\)?

\[ \frac{1}{N_1} \sum_{i=1}^N T_i Y_i - \frac{1}{N_0} \sum_{i=1}^N (1 - T_i) Y_i \]

• This implies \[ {\mathbb{E}}\{Y_i(1)\} \neq {\mathbb{E}}\{Y_i(1) {\:\vert\:}T_i = 1\}, \quad {\mathbb{E}}\{Y_i(0)\} \neq {\mathbb{E}}\{Y_i(0) {\:\vert\:}T_i = 0\} \]

  • e.g., people who read newspapers are more interested in politics.

• Without randomization, treatment and control groups are different with respect to pre-treatment covariates.

• Pre-treatment covariates: Variables that are not affected by the treatment.

• Importantly, potential outcomes are pre-treatment covariates!

• Note: observed outcomes are post-treatment covariates!

• Intuition: Randomization makes treatment and control groups similar on average with respect to all observed and unobserved pre-treatment covariates.

• Treatment variables \((T_1, \ldots, T_N)\) are random.

• Units and potential outcomes (\(Y_i(1), Y_i(0)\)) are fixed.

• We now distinguish Sample Average Treatment Effect (SATE):

  \[ \tau_{SATE} \equiv \frac{1}{N} \sum_{i=1}^N \{Y_i(1) - Y_i(0)\} \]

• Randomization is the “reason basis for inference” (Fisher 1936)

• Design-based inference:

  • Advantage: Rely only on the treatment assignment mechanism that researchers control instead of untestable distributional assumptions (e.g., i.i.d data, normal errors, outcome models).
  • Disadvantage: sometimes less flexible

• Unbiased for the SATE under complete randomization – no modeling assumption is needed.

• First suppose, \(\mathcal{O}_N = \{Y_i(1), Y_i(0)\}_{i=1}^N\), then:

• Can prove that if probabilities of assignment are fixed, i.e. \(\forall i:\: p_i = p\), we have \({\mathbb{E}}[ \widehat{\tau}_{IPW} {\:\vert\:}\mathcal{O}_N ] = \tau_{SATE}\). (How would you approach this?) (proof)

• This estimator is more general:

  1. DiM is a special case of IPW estimator when… \(\forall i:\: p_i = N_1/N\). Note: \(p_i\)’s do not have to be \(0.5\)!
  2. IPW allows to account for complex designs with unequal probabilities of assignment across units.
  3. IPW estimator will show up in observational studies as well.

• This is analogous to simple random sampling without replacement (SRSWOR) from a finite population

• For SRSWOR of \(n\) units from a population of \(N\), the variance of the sample mean is:

\[ {\mathbb{V}}[\overline{Y}] = \frac{S^2}{n} \underbrace{\frac{N - n}{N}}_{FPC} \]

• Intuition: The factor \(\frac{N-n}{N} = 1 - \frac{n}{N}\) reduces variance because sampling w/o replacement removes variability by removing units from population

  • When \(n \ll N\): FPC \(\approx 1\) (negligible correction)
  • When \(n = N\): FPC \(= 0\) (no randomness left – census!)

• Conservative estimator: \({\mathbb{E}}[\widehat{\sigma}^2 {\:\vert\:}\mathcal{O}_N ] \geq {\mathbb{V}}[\widehat{\tau}_{DiM} {\:\vert\:}\mathcal{O}_N]\)

• This estimator achieves its highest value, i.e. \({\mathbb{V}}[\widehat{\tau}_{DiM} {\:\vert\:}\mathcal{O}_N]\) when… treatment effects are constant across units.

• Leads to conservative inferences:
  • Standard errors, \(\widehat{\sigma}\), will be in expectation at least as big as they should be.
  • Confidence intervals using \(\widehat{\sigma}\) will be in expectation at least wide as they should be.
  • Type I (false-positives) error rates will still be correct, power will be lower.

• Sampling introduces an additional layer of uncertainty in causal inference:

  1. \(N\) units are randomly sampled from the population.
  2. \(N_1\) units are then randomly assigned to the treatment.

  \(\rightsquigarrow\) Focus on Population ATE.

• How does this affect our inference, in terms of:

  • Point estimates: Is observed difference in means (\(\widehat{\tau}\)) still unbiased?
  • Uncertainty estimates: Is the conventional standard error still valid?

• Population Average Treatment Effect (PATE) is:

  \[ \tau_{PATE} \equiv {\mathbb{E}}[Y_i(1) - Y_i(0)] \]

• DiM is unbiased (over repeated sampling and treatment assignment):

  \[ {\mathbb{E}}[\widehat{\tau}_{DiM}] = {\mathbb{E}}\left[ {\mathbb{E}}[\widehat{\tau}_{DiM} {\:\vert\:}\mathcal{O}_N] \right] = {\mathbb{E}}[\tau_{SATE}] = {\mathbb{E}}[Y_i(1) - Y_i(0)] = \tau_{PATE} \]

  • Note: This requires a true random sampling from the population.

• Important: Often obtaining such a sample is impossible \(\rightsquigarrow\) External Validity

  • In such a case: focus on SATE and interpret as such (estimate is still internally valid, but no longer externally valid)

• Applying the LTV and denoting the population variance of \(Y_i(t)\) and \(\tau\) as \(\sigma_{t}^2\) and \(\sigma_{\tau}^2\):

  \[ \begin{align*} {\mathbb{V}}[\widehat{\tau}_{DiM}] &= {\mathbb{E}}\left[{\mathbb{V}}[\widehat{\tau}_{DiM} {\:\vert\:}\mathcal{O}_N]\right] + {\mathbb{V}}\left[{\mathbb{E}}[\widehat{\tau}_{DiM}\mid \mathcal{O}_N]\right] \\ &= {\mathbb{E}}\left[\frac{S_{Y_1}^2}{N_1} + \frac{S_{Y_0}^2}{N_0} - \frac{S_{\tau}^2}{N}\right] + {\mathbb{V}}[\tau_{SATE}] \\ &= {\mathbb{E}}\left[\frac{S_{Y_1}^2}{N_1} + \frac{S_{Y_0}^2}{N_0} - \frac{S_{\tau}^2}{N}\right] + {\mathbb{V}}\left[\frac{1}{N}\sum_{i \in \mathcal{O}_N}\tau_i\right] \\ &= {\mathbb{E}}\left[\frac{S_{Y_1}^2}{N_1} + \frac{S_{Y_0}^2}{N_0} - \frac{S_{\tau}^2}{N}\right] + \frac{\sigma_{\tau}^2}{N} \\ &= \frac{\sigma_{1}^2}{N_1} + \frac{\sigma_{0}^2}{N_0}. \end{align*} \]

• This is in contrast to the result for the SATE:

  \[ {\mathbb{E}}[\widehat{\sigma}^2 {\:\vert\:}\mathcal{O}_N] \geq {\mathbb{V}}[\widehat{\tau}_{DiM} {\:\vert\:}\mathcal{O}_N] \]

• Intuition: This variance estimator was always too large for SATE–we overestimated the variability.

• But for PATE, because we have additional uncertainty, this becomes an unbiased estimator.

  • Further reading: Imbens and Rubin (2015, Ch. 6).

• PATE

  • Consistency via the law of large numbers: \(\widehat{\tau}_{DiM} \xrightarrow{p} \tau_{PATE}\).
  • Asymptotic normality via the CLT

  \[ \frac{\widehat{\tau}_{DiM}- \tau_{PATE}}{\sqrt{\sigma^2_1/N_1 + \sigma_0^2/N_0}} \xrightarrow{d} \mathcal{N}(0, 1) \]

• SATE

  • Finite population CLT (Li and Ding 2017).

  \[ \frac{\widehat{\tau}_{DiM}- \tau_{SATE}}{\sqrt{S^2_1/N_1 + S_0^2/N_0 - S^2_{10}/N}} \xrightarrow{d} \mathcal{N}(0, 1) \]

• \((1 - \alpha) \times 100\)% CI: \([\widehat{\tau}_{DiM} - \widehat{\sigma} \times z_{1 - \alpha/2}, \widehat{\tau}_{DiM} + \widehat{\sigma} \times z_{1 - \alpha/2}]\)

• In practice in a completely randomized experiment:

  1. Regress \(Y_i\) on \(T_i\) (w/ intercept) and get the coefficient on \(T_i\).
  2. Calculate the robust standard error (estimatr::lm_robust()).
  3. Calculate confidence intervals, etc. as usual.

• Lady Tasting Tea Experiment

  • Units: 8 identical cups

  • Randomization: Randomly choose 4 cups into which the tea is poured first, and for the other four, the milk was poured first

  • Null hypothesis: the lady cannot tell the difference

  • Statistic: the number of correctly classified cups

  • Outcome: The lady classified all 8 cups correctly!

• Did this happen by chance?

• Treatments are assigned with some known treatment assignment mechanism
  • If researchers can reproduce it, fine to be very complicated
  • e.g., complete, Bernoulli, Block, Cluster, or any complex randomization

• “Sharp” null hypothesis of no treatment effect: \[ H^{sharp}_0: Y_i(1) = Y_i(0) \: \forall i \quad \text{vs.} \quad H^{sharp}_a: \exists i:\: Y_i(1) \neq Y_i(0) \]

• Very different from the “weak” null hypothesis!

• Example: GOTV experiments:

• Estimate: \(\widehat{\tau} = \frac{2}{3} - \frac{1}{2} = \frac{1}{6}\)

• Is this statistically significant? How do we compute \(p\)-value?

• The null (\(\approx\) sampling) distribution of the test statistic is \(\{\widehat{\tau}_k\}_{k=1}^K\), where \[ \widehat{\tau}_k = \frac{\sum_{i=1}^N \widetilde{T}^k_i Y_i}{\sum_{i=1}^N \widetilde{T}^k_i} - \frac{\sum_{i=1}^N (1-\widetilde{T}^k_i) Y_i}{\sum_{i=1}^N (1-\widetilde{T}^k_i)} \]

• Exact (two-sided) \(p\)-value is \(p = \frac{1}{K} \sum_{k=1}^K \mathbb{1} \left[ |\widehat{\tau}_k| > |\widehat{\tau}|\right]\), where \(\widehat{\tau}\) is the observed test statistic

• Note: If \(K\) (the number of potential treatment assignment) is large, use simulations!

2. Difference-in-Mean-Ranks (for continuous outcomes):

   \[ S = \left|\frac{\sum_{i=1}^N T_i R_i}{\sum_{i=1}^N T_i} - \frac{\sum_{i=1}^N (1-T_i) R_i}{\sum_{i=1}^N (1-T_i)} \right| \]

   • Rank of the outcome for unit \(i\): \(R_i = {\mathrm{rank}}(Y_1(T_1), \ldots, Y_N(T_N))\).
   • Advantages: Reference distribution does not depend on scale and is not sensitive to outliers.

• To learn more read Imbens and Rubin (2015, Ch. 5).

2. Choose a test statistic \(S = f(\{Y_i, T_i, \tau_{0i}\}_{i=1}^N)\)

   • Difference-in-Means, Difference-in-Mean-Ranks, etc.
   • Any statistic gives a valid and exact \(p\)-value but power may differ
   • Could use regression models or machine learning algorithms

3. Compute the reference distribution and \(p\)-value based on the randomized distribution of treatment assignment
   • Exact distribution in small samples
   • Monte Carlo approximation as a general strategy

• Ho and Imai (2006) rely on data from the 2003 CA Gubernatorial Recall Election

  • 135 candidates.
  • Ballot order differs across 80 districts.
  • assembly district-counties that have more than one page ballots.

• Setup:

  • Null hypothesis: no causal effect of being on first ballot page for any candidate.
  • Test statistic: DiM between being on the first ballot page and being on other ballot pages.
  • Reference (randomization) distribution via Monte Carlo simulations.

• Extensions

  • Can be extended to deal with non-sharp null, e.g. in the interference setting (Athey, Eckles, and Imbens 2018).

• But
  • In small samples, you may get unlucky and suffer from imbalance.
  • In a “natural” randomized experiment, it’s important to check whether randomization occurred as you thought.

• Common practice: Conduct balance checks with respect to observed pre-treatment covariates.

  • Compare means, standard deviations, etc., between the treated and untreated.
  • Can also regress treatment indicator on covariates.
  • Visual inspection of histograms/density plots.
  • Many packages have balance tests, e.g. RItools::xbalance().

• Potential benefit: If \(X_i\) predicts \(Y_i\), including it as covariate, \(Y_i = \alpha + \tau T_i + X_i^\prime \gamma\), reduces residual variance \(\Rightarrow\) smaller SEs \(\Rightarrow\) precision gain.

• Pitfall 1 — Overfitting: In cases where \(X_i\) is weakly predictive of \(Y_i\), naive adjustment introduces additional noise and can even bias estimates in small samples (Freedman 2008).

• Pitfall 2 — Common-slope constraint: Naive adjustment forces parallel regression lines with respect to \(X\) and limits precision gains.

  • Natural fix is full interaction model, \(Y_i = \alpha + \tau T_i + X_i^\prime \gamma + T_i X_i^\prime \xi\), to allow different slopes. But this induces \({\mathrm{cov}}(T_i, T_i \tilde{X}_i) \neq 0\) and \(\tau\) is the treatment effect at \(X = 0\), not the ATE.

• This addresses both pitfalls:

  1. Interactions \(\Rightarrow\) remove common-slope constraint \(\Rightarrow\) flexible fit
  2. Demeaning \(\Rightarrow\) \(\tau\) is the effect at \(\overline{X}\) (\(\approx\) ATE); \({\mathrm{cov}}(T_i, T_i \tilde{X}_i) = 0\) by construction. (proof)
  3. Variance guarantee: asymptotic variance of \(\widehat{\tau}_{Lin}\) \(\leq\) variance of DiM \(\Rightarrow\) you can only gain precision, never lose it.
  4. Bias is \(O(1/N)\) and decreases with sample size \(\Rightarrow\) consistent; use HC2 robust SEs for valid inference.

• Further reading: Lei and Ding (2021) propose a procedure to debias Lin’s estimator even in small samples.

• Problem: Unequal probabilities can bias the Difference-in-Means (DiM) estimator.

• Solution: Use the Inverse Probability Weighting (IPW) that we introduced before estimator to account for unequal probabilities.

• The expected value of the DiM estimator is:

\[ \begin{align*} {\mathbb{E}}[\widehat{\tau}] &= {\mathbb{E}}\left[ \frac{1}{N_1}\sum_{i=1}^N T_i Y_i(1) \right] - {\mathbb{E}}\left[ \frac{1}{N_0}\sum_{i=1}^N (1 - T_i) Y_i(0) \right] \\ &= \sum_{i=1}^N {\mathbb{E}}\left[\frac{T_i}{N_1}\right] Y_i(1) - \sum_{i=1}^N {\mathbb{E}}\left[\frac{(1 - T_i)}{N_0}\right] Y_i(0) \\ &= \frac{\sum_{i=1}^N p_i\,Y_i(1)}{\sum_{j=1}^N p_j} - \frac{\sum_{i=1}^N (1-p_i)\,Y_i(0)}{\sum_{j=1}^N (1-p_j)} \\ &\neq \tau_{ATE}. \end{align*} \]

• Intuition: The DiM over-weights observations with higher assignment probabilities and is generally not equal to the true ATE (unless \(p_i\) is constant for all units or \(\tau_i\) is constant).

• Properties:

  • The IPW estimator is unbiased for the ATE: \({\mathbb{E}}[\widehat{\tau}_{IPW}] = \tau_{ATE}\)
  • Can be used for estimation of SATE or PATE.
  • Can be obtained using regression with inverse probability weights (e.g. estimatr::lm_robust() with weights argument).

• Intuition: Weight each unit by the inverse of its probability of receiving the observed treatment to compensate for over-weighting.

• Note: The IPW estimator is unbiased and consistent if the treatment assignment mechanism is known and correctly specified.

• Standard errors ignoring cluster randomization are usually too small (opposite of conservative).

• This is due to units within the same cluster typically being more similar than units in different clusters.

• To address the issue, as before, we can:

  • Use weighted estimators, such as the IPW estimator, or
  • Regression weighted by inverse of probabilities of assignment to account for unequal cluster sizes.

• Then we can define the intracluster correlation: \(\rho = \frac{\sigma^2_B}{\sigma^2} = 1 - \frac{\sigma^2_W}{\sigma^2}\)

• Intuition: When \(\rho\) is \(1\) (\(0\)), responses are identical (uncorrelated) within each cluster.

• Valid inference:

  • OLS and DiM estimators are unbiased for \(\tau_{ATE}\) if clusters are equally sized.
  • Possible bias if cluster sizes vary and are correlated with potential outcomes.
  • Rely on cluster-robust standard errors (CR2) or randomization inference.

• Note: When \(G\) is small, \(\rho\) will be poorly estimated and cluster SEs will be unreliable \(\rightsquigarrow\) prefer increasing \(G\) over sample size per cluster (\(N_j\)).

• Example: \(n = 4\) with two males and two females.

  • Complete randomization will place two females in the same treatment group \(\frac{1}{3}\) of the time.

  • If that happens, how can we tell the treatment effect from gender difference?

• Solution: Pre-stratify the sample, and then randomize completely within each stratum

  1. Blocking will perfectly balance \(X_i\).
  2. Randomization will balance the rest in expectation.

  Note

  \(\rightsquigarrow\) “Block what you can; randomize what you cannot.” (George Box)

• SATE within blocks is defined by \(V_i\):

\[ \tau_\text{v} = \frac{1}{N_\text{v}} \sum_{i:V_i=1} [ Y_i(1) - Y_i(0) ], \qquad \tau_\text{nv} = \frac{1}{N_\text{nv}} \sum_{i:V_i=0} [ Y_i(1) - Y_i(0) ] \]

• Using Law of Iterated Expectation:

\[ \tau_{SATE} = \underbrace{\left( \frac{N_\text{v}}{N_\text{v} + N_\text{nv}} \right)}_{\text{share voters}} \tau_\text{v} + \underbrace{\left( \frac{N_\text{nv}}{N_\text{v} + N_\text{nv}} \right)}_{\text{share non-voters}} \tau_\text{nv} \]

• Probability of treatment in each group called the propensity score:

  • Prob. of treatment for voters: \({\textrm{Pr}}(T_i = 1 {\:\vert\:}V_i = 1) = p_{\text{v}} = \frac{N_{1,\text{v}}}{N_{\text{v}}}\).
  • Prob. of treatment for non-voters: \({\textrm{Pr}}(T_i = 1 {\:\vert\:}V_i = 0) = p_{\text{nv}} = \frac{N_{1,\text{nv}}}{N_{\text{nv}}}\).

• Blocking ensures balance across blocks:

  • When \(p_{\text{v}} = p_{\text{nv}}\), distribution of treatment is exactly the same in each block.
  • With complete randomization, treatment might be very imbalanced (in absolute terms) across \(V_i\).
  • Benefit: No possibility of chance imbalances skewing the estimates.

• Unbiased estimator for the overall SATE for block design:

\[ \widehat{\tau}_{BR} = \left(\frac{N_\text{v}}{N}\right) \widehat{\tau}_\text{v} + \left(\frac{N_\text{nv}}{N}\right) \widehat{\tau}_\text{nv} \]

• Property: Equivalent to the regular DiM if \(p_\text{v} = p_\text{nv} = \frac{1}{2}\).
• Otherwise, standard \(\widehat{\tau}_{DiM}\) under block design will be biased.

• Finite sample variance of the blocked (BR) estimator:

\[ {\mathbb{V}}[\widehat{\tau}_{BR} {\:\vert\:}\mathcal{O}] = \left(\frac{N_\text{v}}{N}\right)^2 {\mathbb{V}}[\widehat{\tau}_\text{v} {\:\vert\:}\mathcal{O}] + \left(\frac{N_\text{nv}}{N}\right)^2 {\mathbb{V}}[\widehat{\tau}_\text{nv} {\:\vert\:}\mathcal{O}] \]

• Use the conservative variance estimators from each strata to get:

\[ \widehat{\sigma}_{BR} = \left(\frac{N_\text{v}}{N}\right)^2 \left(\frac{\widehat{\sigma}^2_{1,\text{v}}}{N_{1,\text{v}}} + \frac{\widehat{\sigma}^2_{0,\text{v}}}{N_{0,\text{v}}}\right) + \left(\frac{N_\text{nv}}{N}\right)^2 \left(\frac{\widehat{\sigma}^2_{1,\text{nv}}}{N_{1,\text{nv}}} + \frac{\widehat{\sigma}^2_{0,\text{nv}}}{N_{0,\text{nv}}}\right), \]

where \(\widehat{\sigma}^2_{t,\text{v}}\) are the within-strata observed outcome variances.

• Within-block estimators:

\[ \begin{align*} \widehat{\tau}_j &= \frac{1}{N_{1,j}} \sum_{i:B_i=j} T_i Y_i - \frac{1}{N_{0,j}} \sum_{i:B_i=j} (1 - T_i) Y_i, \\ \widehat{\sigma}_j &= \frac{\widehat{\sigma}^2_{1,j}}{N_{1,j}} + \frac{\widehat{\sigma}^2_{0,j}}{N_{0,j}} \end{align*} \]

• Aggregate blocking estimators:

\[ \widehat{\tau}_{BR} = \sum_{j} w_j \widehat{\tau}_j, \qquad \widehat{\sigma}_{BR} = \sum_{j} w_j^2 \widehat{\sigma}(\widehat{\tau}_j) \]

• Finite sample difference in sampling variances can be given by \(\widehat{\sigma}_{R} - \widehat{\sigma}_{BR} = \frac{1}{n-1} (B - W)\), where measures of between- and within-block variation:

\[ \begin{align*} B &= \sum_{j=1}^{J} \left(\frac{N_j}{N}\right) (\overline{Y}_j(1) + \overline{Y}_j(0) - (\overline{Y}(1) + \overline{Y}(0)))^2 \\ W &= \sum_{j=1}^{J} \frac{N_j}{N} \frac{N_{1,j} N_{0,j}}{N_j} \widehat{\sigma}(\widehat{\tau}_j {\:\vert\:}\mathcal{O}) \end{align*} \]

• Difference can be positive or negative (Pashley and Miratrix 2022):

  • Intuition: Blocking is better when outcomes vary a lot across blocks, not much within blocks (blocks are predictive of outcomes, so usually the case).
  • Blocking is also more efficient for PATE under stratified sampling.

• \(p_j = p\): Use OLS with block dummies (or fixed effects) to get an unbiased estimate of the ATE:

\[ Y_i = \alpha + \tau T_i + \sum_{j=2}^{M} \beta_j B_{ij} + \epsilon_i, \quad \text{where} \quad {\mathbb{E}}[\widehat{\tau}_{OLS}] = \tau \]

• Valid uncertainty estimates can then be obtained via HC2 standard errors (or clustered SE if randomization was clustered within blocks).

• \(p_j\) varies by block: Use weighted least squares instead of OLS, where the weight is the inverse probability of treatment/control for \(i\) in block \(j\):

\[ \forall i, j:\: w_{ij} = \begin{cases} \frac{1}{p_j}, & \text{if } T_i = 1 \\ \frac{1}{(1 - p_j)}, & \text{if } T_i = 0 \end{cases} \]

• Statistical power of a design + test \(\equiv\) the test’s probability of rejecting the null in favor of the alternative when the alternative is indeed true, given the design.

• Example: \(H_0:\: \beta = 0\) and \(H_a:\: |\beta| = \tau > 0\), giving rise to a two sided test. Then power is given by, \(\kappa = {\textrm{Pr}}[\text{Reject } H_0 {\:\vert\:}H_a \text{ is true; design, test}]\).

• What does power depend on?

  • True size of the effect (\(\tau\)),
  • Sample size and proportion of the treated (\(N\) and \(p\)),
  • Variability of potential outcomes (\(\sigma\)),
  • Test statistic,
  • Number of treatments,
  • Randomization scheme (simple, complete, clustered, blocked, etc.)

• Recall that our asymptotically valid variance expression is:

\[ {\mathbb{V}}(\hat\tau) = \frac{\sigma^2_1}{p N}+\frac{\sigma^2_0}{(1-p) N} \]

• How should we proceed? Solve a minimization problem!

• Then, for the \(t\)-statistic of equality of means, we have:

\[ \frac{\widehat{\tau}_{DiM}-\tau}{\sqrt{\frac{\sigma^2_1}{N_1}+\frac{\sigma^2_0}{N_0}}} = \frac{\widehat{\tau}_{DiM}-\tau}{\sqrt{\frac{2\sigma^2}{N}+\frac{2\sigma^2}{N}}} = \frac{\widehat{\tau}_{DiM}-\tau}{2\sigma/\sqrt{N}} \sim \mathcal{N} (0,1). \]

• Therefore:

\[ t = \frac{\widehat{\tau}_{DiM}}{\sqrt{\frac{\sigma^2_1}{N_1}+\frac{\sigma^2_0}{N_0}}} \sim \mathcal{N}\left(\frac{\tau\sqrt{N}}{2\sigma},1\right) \]

• To choose \(N\) we need to specify:

  1. \(\tau\): usually \(0.25 \sigma_0\),
  2. Target power value (\(1- \psi\)): usually 0.80 or higher,
  3. \(\sigma_1^2\) and \(\sigma_0^2\): e.g. using previous measures,
  4. \(p\): proportion of observations in the treatment group

• Testing \(H_0 : \beta = 0\) relative to \(H_a : |\beta| = \mu > 0\).

• Large-sample distribution of \(\hat{\beta}\) under \(H_0\), denoted by \(F_{\hat{\beta}}\bigl(\cdot \mid \beta = 0\bigr)\).

• For a test at \(100\bigl(1 - \alpha\bigr)\%\) confidence, the distribution under \(H_0\) defines the “rejection region.”

• Large-sample distribution of \(\hat{\beta}\) under \(H_a\), denoted by \(F_{\hat{\beta}}\bigl(\cdot {\:\vert\:}\beta = b_{H_a}\bigr)\).

• Power is the probability of falling in the “rejection region”:

  \[ \kappa = 1 - F_{\hat{\beta}}\bigl(t_{\alpha/2} \sigma_{\hat{\beta}} \mid \beta = b_{H_A}\bigr) \]

• For a test with power \(\kappa\), need \(b_{H_a}\) as in the picture.

• Suppose we test against \(\mathcal{N} (0,1)\) with \(\alpha = 0.05\) and \(\kappa = 0.80\). These are the standards in social science. Then We have, \(t_{\alpha/2} + t_{1-\kappa} = |z_{.025}| + |z_{.2}| = 1.96 + 0.84 = 2.80\).

• Therefore, the MDE is 2.8 times the conservative standard error of the effect estimator for any study for which we use

  1. a two-sided test,
     
  2. a standard normal distribution as our reference distribution,
     
  3. \(\alpha = .05\) and \(\kappa = .80\)

• In all applications we have studied, the normal or \(t\) distribution is a good approximation so long as the sample size is not too small.

• Plug into the expression for MDE to get sample size determination formula.

\[ \begin{align*} MDE &= \bigl(t_{\alpha/2} + t_{1-\kappa} \bigr) \sqrt{\frac{1}{N} \left( \frac{\sigma^2_{1}}{p} + \frac{\sigma^2_{0}}{1 - p} \right)} \\ \implies N &= \frac{\bigl(t_{\alpha/2} + t_{1-\kappa} \bigr)^2 \left( \frac{\sigma^2_{1}}{p} + \frac{\sigma^2_{0}}{1 - p} \right)}{MDE^2}. \end{align*} \]

• Note Use standardized effect sizes to avoid “power fallacy”!

  • E.g., Effect size relative to the control group standard deviation, Glass’s \(\Delta\), \(\Delta = MDE / \sigma_{0}\) (normalize standard deviations too!)

• Optimal design considerations:

  • Balance treatment and control groups (\(p \approx 0.5\)) if you do not expect differences in potential outcomes variability.
  • Increase sample size to detect smaller effects.
  • Use blocking or stratification to improve precision.

• Practical considerations:

  • Ensure sufficient power to detect meaningful effects.
  • Consider trade-offs between sample size and costs/harm.
  • Use power analysis to plan experiments effectively (e.g. EGAP’s power calculators).

• Further reading: Blair, Coppock, and Humphreys (2023, Ch. 10-11).

• Weighting Methods: Utilize inverse probability weighting (IPW) and propensity score weighting to reweight the study sample to mirror the target population. Methods include stabilized weights and entropy balancing.

• Sensitivity Analysis: Perform sensitivity analyses using methods like Rosenbaum bounds, \(E\)-values, and tipping point analysis to evaluate the robustness of findings to potential assumption violations.

• Multilevel Modeling: Apply hierarchical models such as mixed-effects models, random effects models, and Bayesian hierarchical models to account for clustering and variation at different levels (e.g., individual, group, context).

• External Validation: Replicate the study in various settings using techniques like cross-validation, split-sample validation, and external dataset validation to confirm the generalizability of the findings.